Asymmetric Camera Sensor Positioning for Enhanced Package Detection

ABSTRACT

This document describes asymmetric camera sensor positioning for enhanced package detection. In aspects, an electronic doorbell has an image sensor that is rotated to a portrait orientation and vertically shifted relative to a lens of a camera, resulting in asymmetric positioning of the image sensor relative to the lens. The lens projects an image circle onto the image sensor and the image sensor has a sensor detection area having upper corners within the image circle and lower corners outside of the image circle to enable capture of an object located in a lower portion of the image circle and proximate to an edge of the image circle. Then, lens distortion correction is performed on a captured image to provide a final image usable to detect the package, which may be located within the image circle but outside of a conventional sensor detection area.

RELATED APPLICATION

This application is a continuation of, and claims priority to, PCTApplication Number PCT/US2021/044190, filed on Aug. 2, 2021 which isincorporated herein by reference in its entirety.

BACKGROUND

With advances in electronic doorbells for capturing images and/orvideos, many users have begun to rely on their doorbell data todetermine if a package has been delivered. However, many existingelectronic doorbells have cameras with a limited field of view (FOV).Generally, the doorbell is oriented to enable image capture of aperson's face, but not necessarily that person's feet, because the usermay be more interested (for security) in seeing the person's face. Inmany instances, if the package is delivered too close to the doorbell(e.g., placed on the ground under the doorbell and next to the wall onwhich the doorbell is mounted), the package may be outside of thecamera's FOV.

The user may receive a notification of a delivery and retrieve imagedata of a delivery driver arriving and departing, but if the user wishesto check on the status of their delivered package, the user may not beable to see the package in the doorbell data if the package is outsideof the camera's FOV (e.g., if the package was placed too close to thedoorbell). Further, package detection algorithms can be applied to thedoorbell data, but if the package is outside the camera's FOV, thepackage cannot be detected in the captured images, and the user may notbe notified of the package. If the package is not in the camera's FOVand a person approaches to take or steal the package, then the user maynot be notified that the package has been taken.

One solution to expanding the camera's FOV is to use a doorbell camerawith a 180° angle of view (AOV). However, such cameras increasemanufacturing costs and may require additional architectural features toprevent infrared (IR) flare. Using such a camera may also provideadditional image data (e.g., pixel data) to be displayed on the user'sdevice that may be unimportant, including an area above the person'shead. Because a finite number of pixels exist that can be displayed inan application on a screen (e.g., a smartphone's display) as the user isviewing the doorbell data, the unimportant additional image dataconsumes screen real estate and may result in a smaller displayed image,decreased image quality, and diminished user experience.

SUMMARY

This document describes asymmetric camera sensor positioning forenhanced package detection. In aspects, an electronic doorbell has animage sensor that is rotated to a portrait orientation and verticallyshifted relative to a lens of a camera, resulting in asymmetricpositioning of the image sensor relative to the lens. The lens projectsan image circle onto the image sensor, and the image sensor has a sensordetection area having upper corners within the image circle and lowercorners outside of the image circle to enable capture of an objectlocated in a lower portion of the image circle and proximate to an edgeof the image circle. Then, lens distortion correction is performed on acaptured image to provide a final image usable to detect the package,which may be included within the image circle but outside of aconventional sensor detection area.

In some aspects, an electronic doorbell is disclosed. The electronicdoorbell includes a lens having a lens optical axis and providing animage circle representing a scene captured by the lens. The image circlehas an upper portion and a lower portion separated by a middle portionand arranged in a vertical stack. The upper portion is located proximateto an upper edge of the image circle and the lower portion is locatedproximate to a lower edge of the image circle. The electronic doorbellalso includes an image sensor having a sensor detection area. The imagesensor is positioned in a portrait orientation relative to the verticalstack of portions of the image circle, the portrait orientation of thesensor detection area having a vertical dimension that is greater than ahorizontal dimension. Also, the image sensor is vertically shiftedrelative to the lens by an offset distance from the lens optical axis toenable the image sensor to capture an image of an object located in thelower portion of the image circle proximate to the lower edge of theimage circle.

This summary is provided to introduce simplified concepts concerningasymmetric camera sensor positioning for enhanced package detection,which is further described below in the Detailed Description andDrawings. This summary is not intended to identify essential features ofthe claimed subject matter, nor is it intended for use in determiningthe scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of one or more aspects of asymmetric camera sensorpositioning for enhanced package detection are described in thisdocument with reference to the following drawings. The use of the samereference numbers in different instances in the description and thefigures indicate similar elements:

FIG. 1 illustrates an example implementation of asymmetric camera sensorpositioning for enhanced package detection in comparison to aconventional camera doorbell;

FIG. 2 illustrates an isometric view of an example implementation of theelectronic doorbell from FIG. 1 ;

FIG. 3 illustrates a sectional view of the electronic doorbell from FIG.2 , taken along line 3-3, and an enlarged view of a camera-side end ofthe electronic doorbell in the sectional view;

FIG. 4 illustrates an example implementation of the sensor detectionarea in relation to the image circle based on asymmetric positioning ofthe image sensor relative to the lens;

FIG. 5 illustrates an example implementation of the sensor detectionarea shifted relative to the image circle for enhanced packagedetection;

FIG. 6 illustrates an example implementation of lens distortioncorrection in accordance with asymmetric camera sensor positioning in anelectronic doorbell;

FIG. 7 depicts an example method for correcting lens distortion of animage captured by an asymmetrically positioned camera sensor, inaccordance with the techniques described herein; and

FIG. 8 is a block diagram illustrating an example system that includesan example device, which can be implemented as any electronic device(e.g., the electronic doorbell) that implements aspects of asymmetriccamera sensor positioning as described with reference to FIGS. 1-7 .

DETAILED DESCRIPTION Overview

This document describes asymmetric camera sensor positioning forenhanced package detection. The techniques described herein provide anelectronic doorbell camera with a greater vertical FOV in comparison toconventional doorbell cameras. For example, the electronic doorbellcamera includes an image sensor that is rotated to a portraitorientation and also vertically offset from the lens of the camera. Thisasymmetric positioning enables the image sensor to capture an image ofan object (e.g., a package) located on the ground substantially belowthe electronic doorbell camera, where the object may be captured by aconventional lens but not sufficiently projected by the lens onto aconventional image sensor. Lens distortion in the captured image is thencorrected in post-processing to provide a clean final image to a user,where the image may include a standing person's face as well as thepackage on the ground.

Not only is the vertical FOV increased, but the number of useful pixelsto the user is increased, resulting in increased image quality andefficiency and an enhanced user experience. Lens distortion associatedwith objects near the corners of the image is reduced. Further, theoverall efficiency of using a standard 160° lens is increased. Whilefeatures and concepts of the described asymmetric camera sensorpositioning for enhanced package detection can be implemented in anynumber of different environments, aspects are described in the contextof the following examples.

Example Device

FIG. 1 illustrates an example implementation of asymmetric camera sensorpositioning for enhanced package detection in comparison to aconventional doorbell camera. Many conventional doorbell cameras captureimages with a wide horizontal field of view (hFOV) and a short verticalfield of view (vFOV), which enable image capture of people but notnecessarily objects (e.g., packages) on the ground. Example 100illustrates a conventional doorbell camera 102 mounted to a wall 104(represented by a vertical dashed line) and having a vertical AOV 106that is approximately 104° and oriented to capture a head and face 108of a person 110 having a height of approximately 6 feet (ft) 2 inches(in) (1.88 meters (m)) and standing at a horizontal distance 112 ofapproximately 2 ft (0.6 m) from the conventional doorbell camera 102. Inthis example, the conventional doorbell camera 102 is unable to capturean image of a package 114 located in a region (e.g., region 116) that issubstantially below the conventional doorbell camera 102 and itsvertical AOV 106 (e.g., a volume between the edge of the vertical AOV106, the conventional doorbell camera 102, the ground, and the wall104). In the example illustration, the package 114 is located at adistance 118 of approximately 6 in (0.15 m) from the wall 104 and is“hidden” from the conventional doorbell camera 102.

Example 120 illustrates an electronic doorbell 122 with asymmetriccamera sensor positioning, as described herein. The electronic doorbell122 includes a vertical AOV 124 of approximately 130°, which provides acorresponding vFOV that enables image capture of the head and face 108of the person 110 described above as well as the package 114 located onthe ground. Using a greater vertical AOV (e.g., vertical AOV 124)reduces the size of the region 126, in comparison to the conventionalvertical AOV 106, increases the corresponding vFOV for image capture,and prevents the package 114 from “hiding.” To enhance the vertical AOV124 of the electronic doorbell 122 over the vertical AOV 106 of theconventional doorbell camera 102, the electronic doorbell 122 includes acamera sensor (e.g., image sensor 128) that is oriented in a portraitorientation and asymmetrically aligned with a camera lens (e.g., lens130) of the electronic doorbell 122, resulting in an sensor detectionarea 132 that is offset (e.g., not optically centered) from an imagecircle 134. In particular, the image sensor 128 is not opticallycoaxially aligned with the lens 130 but is vertically offset. In thisway, the electronic doorbell 122 can achieve a greater vertical AOV (anda greater vFOV) than the conventional doorbell camera 102 by usingsimilar components and architecture and without implementing a moreexpensive image sensor and/or lens. The image circle 134 is a crosssection of a cone of light transmitted by the lens 130 onto the imagesensor 128. The sensor detection area 132 of the image sensor 128represents an area of light sensed by the image sensor 128. Generally,the image circle 134 is projected onto the sensor detection area 132 toenable the image sensor 128 to detect at least a portion of the imagecircle 134. Depending on certain factors (e.g., focal length, distancebetween the lens 130 and the image sensor 128, alignment, size of theimage sensor 128), the sensor detection area 132 may capture some or allof the image circle 134 projected onto the image sensor 128. Furtherdetails of these and other features are described below.

FIG. 2 illustrates an isometric view 200 of an example implementation ofthe electronic doorbell 122 from FIG. 1 . The electronic doorbell 122includes a housing 202 having an elongated shape (e.g., substantiallyobround in a front view) with opposing rounded ends intersected by alongitudinal axis 204 of the housing 202. A camera module (e.g., cameramodule 206 having the image sensor 128 and the lens 130 from FIG. 1 ) ispositioned within the housing 202 proximate to a first end (e.g.,camera-side end 208) of the housing 202. A pressable button 210 ispositioned proximate to a second end (e.g., a button-side end 212) ofthe housing 202.

The housing 202 may include a plastic material and be formed, forexample, using plastic-injection molding techniques. The housing 202 mayinclude any suitable geometry, including the example geometryillustrated in FIG. 2 . For instance, the housing 202 may includemultiple components forming a shell (e.g., a hollow, substantiallyobround shell) that fit together (e.g., snap together) to form a cavityto house various components of the electronic doorbell 122. The housing202 may also include an aperture or transparent region that is alignedwith the camera module 206 to enable the camera module 206 to viewthrough the aperture or transparent region and capture images or videoof a scene.

The button 210 may include any suitable button (e.g., a mechanicalbutton to open or close a switch, a capacitive sensor to detect usertouch) usable to initiate a function. For example, actuation of thebutton 210 may initiate a function, including a ringing of an audibledoorbell, transmission of an electronic notification to a smartphone ofthe doorbell's owner, initiation of the camera module 206, and so on.Any suitable function can be initiated by activating the button 210.

FIG. 3 illustrates a sectional view 300 of the electronic doorbell 122from FIG. 2 , taken along line 3-3, and an enlarged view 302 of thecamera-side end 208 of the electronic doorbell in the sectional view300. Within the housing 202, the electronic doorbell 122 includesmultiple printed circuit boards (PCBs), including at least a main logicboard 304 and a camera board 306. Additional PCBs may also be used. ThePCBs may include various integrated circuit (IC) components, includingsystem-on-chip (SoC) IC devices, processors, and IC components forlight-emitting diode(s) (LEDs), microphone(s), or sensors for detectinginput such as touch-input, a button-press, a voice command, or motion.The electronic doorbell 122 also includes the camera module 206 (e.g., acamera), a battery 308, the button 210, and a speaker module 310. Thebattery 308 may be positioned between the camera-side end 208 and thebutton-side end 212.

The speaker module 310 may output audio waves toward a front and/orsides (e.g., lateral sides that are orthogonal to a front surface 312 ofthe housing 202) of the electronic doorbell 122. The speaker module 310can enable a visitor (e.g., a user pressing the button 210) to listen toan audible message, including a recorded audio message or a real-timeaudio transmission from the doorbell's owner.

The battery 308 provides power to the electronic doorbell 122 andenables the electronic doorbell 122 to be wireless. Because theelectronic doorbell 122 is battery powered, the electronic doorbell 122can be mounted in any suitable location without having to hardwire theelectronic doorbell 122 to an electric power source. For example, theelectronic doorbell 122 can be mounted on a user's house proximate totheir front door without having to drill holes in the house to connectwires to a power source inside the house.

The PCBs (e.g., the main logic board 304, the camera board 306) may beformed, for example, from glass-reinforced epoxy material such as FR4.In some instances, the PCBs may include a single layer of electricallyconductive traces and be a single-layer board. In other instances, thePCBs may be a multi-layer board that includes multiple layers ofelectrically conductive traces that are separated by layers of adielectric material.

The electronic doorbell 122 also includes a passive infrared (PIR)sensor 314 positioned within the housing 202 proximate to thecamera-side end 208. The PIR sensor 314 is configured to detect motionof an object (e.g., human or animal) within an FOV of the PIR sensor314.

The camera module 206 includes various components, including the imagesensor 128 and the lens 130. In aspects, the lens 130 has an opticalcenter (e.g., lens optical axis 316 representing a straight line passingthrough the geometrical center of the lens 130 and joining the centersof curvature of the lens's surfaces). The image sensor 128 also has anoptical center (e.g., image-sensor optical axis 318), which inconventional camera systems is typically aligned with the lens opticalaxis 316 of the lens 130. However, in the electronic doorbell 122described herein, the image-sensor optical axis 318 is vertically offsetfrom the lens optical axis 316. Further details are described inrelation to FIG. 4 .

FIG. 4 illustrates an example implementation 400 of the sensor detectionarea 132 in relation to the image circle 134 based on asymmetricpositioning of the image sensor 128 relative to the lens 130 (shown inFIG. 1 ). In aspects, the image sensor 128 is shifted relative to thelens 130 in order to shift the sensor detection area 132 relative to theimage circle 134. In addition, the image sensor 128 is rotated 90degrees to cause the sensor detection area 132 to have a portraitorientation (e.g., a 3:4 portrait orientation). This portraitorientation provides a vertical dimension (e.g., height 402) that isgreater than a horizontal dimension (e.g., width 404) for the imagesensor 128. By implementing both the portrait orientation and the shiftof the image sensor 128, the vertical AOV (and the corresponding vFOV)is increased (e.g., see example 120 in FIG. 1 showing the vertical AOV124 of approximately 130°). Accordingly, the height 402 of the imagesensor 128 enables the image sensor 128 to capture an area of the imagecircle 134 corresponding to a vertical AOV of the lens 130 ofapproximately 130°.

The image circle 134 may include multiple portions, including an upperportion 406 and a lower portion 408 separated by a middle portion 410and arranged in a vertical stack. The upper portion is located proximateto an upper edge 412 of the image circle. The lower portion is locatedproximate to a lower edge 414 of the image circle. In some examples, themiddle portion may be further divided into a left portion 416 and aright portion 418 separated by a center portion 420 and arranged in ahorizontal stack. It is noted that the terms “upper” and “lower” aredescribed relative to the illustrated examples and are not intended tobe limited with respect to a particular orientation of componentsrelative to external factors (e.g., Earth, gravity). The describedtechniques may also be implemented by switching the terms “upper” and“lower” herein and applying the techniques in a lens arrangement withmirror-inversion, in which a projected image of the scene is invertedwhen detected by the image sensor 128.

In aspects, the image circle 134 may be associated with the lens 130having an AOV range of approximately 160° (referred to as a 160° lens)which may be substantially less expensive than a 180° lens. Inconventional camera systems that use a 160° lens, the sensor detectionarea 132 (e.g., having a rectangular shape) may not capture upper andlower portions (e.g., the upper portion 406 and the lower portion 408,respectively) of the image captured in the image circle 134 (e.g.,having an elliptical shape) if all of the corners of the sensordetection area 132 are located within the image circle 134.

In the illustrated example, however, the upper two corners (e.g., uppercorners 422) of the sensor detection area 132 are located inside theelliptical shape (e.g., circular shape) of the image circle 134, and thelower two corners (e.g., lower corners 424) are located outside theboundary of the image circle 134. The image-sensor optical axis 318 ishorizontally aligned (e.g., aligned along a horizontal axis 426) withthe lens optical axis 316 but is vertically shifted (e.g., along avertical axis 428) so as to be vertically asymmetric with the lensoptical axis 316. In an example using a mirror-inversion lensarrangement, the lower corners 424 may be located inside the imagecircle 134 while the upper corners 422 are located outside the imagecircle 134.

This image-sensor optical axis 318 may be shifted relative to the lensoptical axis 316 by any suitable distance (e.g., offset distance 430),which enables the image sensor 128 to capture the lower portion 408 ofthe image circle 134 that is proximate to a boundary (e.g., an edge) ofthe image circle 134. In an example, the offset distance 430 may besubstantially within a range of 0.15 millimeters (mm) to 0.35 mm Inaspects, the upper portion 406 (e.g., area between the upper edge 412 ofthe image circle 134 and a top edge 432 of the sensor detection area132) may not include useful pixels because it generally includes an areaabove the person's head with uninteresting or unimportant image data.Accordingly, locating the upper corners 422 of the sensor detection area132 within the image circle prevents the upper two corner areas of acaptured image from having black pixels. The lower portion 408, however,may include useful pixels because it may include the package 114 (shownin FIG. 1 ) located on the ground in proximity to the wall upon whichthe electronic doorbell 122 is mounted. Therefore, locating the lowercorners 424 outside of the image circle 134 enables capture of the lowerportion 408.

Because a portion of the sensor detection area 132 is beyond the loweredge 414 of the image circle 134 projected onto the image sensor 128,the sensor detection area 132 may include two lower corner areas 434that result in black pixels (e.g., vignetting) in corresponding bottomcorner areas of a captured image. Consequently, the bottom corner areasof an image captured by the image sensor 128 become black but can becorrected using distortion correction techniques. Shifting the imagesensor 128, however, reduces distortion and image artifacts that may beintroduced by distortion correction techniques applied to the image, inparticular with objects near the corners of the image. In aspects, thecenter (e.g., midpoint 436) of the bottom edge (e.g., bottom edge 438between the corners (e.g., lower corners 424) that are outside of theimage circle 134) of the sensor detection area 132 is positioned withinthe image circle 134, rather than being tangent to the edge of the imagecircle 134 or outside of the image circle 134. This positioning of thebottom edge 438 of the sensor detection area 132 relative to the imagecircle 134 reduces the number of black pixels included near the bottomof the captured image and improves the efficiency of the lens distortioncorrection techniques described herein. In another example, however, thecenter of the bottom edge 438 of the sensor detection area 132 may betangent to the edge (e.g., lower edge 414) of the image circle 134. Inyet another example, the midpoint 436 of the bottom edge 438 of thesensor detection area 132 may be located beyond the lower edge 414 ofthe image circle 134 but may result in additional black pixels thatnecessitate some additional post-processing procedures for theirremoval.

Additionally, shifting the sensor detection area 132 relative to theimage circle 134 increases the quality of the image by increasing (e.g.,maximizing) the number of useful pixels for the user. For example,capturing the upper portion 406 and subsequently cropping thecorresponding data reduces image quality because pixels are being“thrown away.” By capturing a finite number of pixels, removing aportion (e.g., the upper portion 406), and then enlarging the resultantimage for display on a user's screen, the remaining portion of the imagemay become blurry or pixilated. Accordingly, by shifting the sensordetection area 132, the image sensor 128 is capturing those parts of theimage that have useful image data and/or are important to the user(e.g., a greater vFOV to capture a person's face as well as a packagelocated on the ground under the electronic doorbell 122). Consequently,shifting the image sensor 128 and performing post-processing on thecaptured image, as described herein, not only increases the vFOV forcapturing an image but also reduces (e g, minimizes) the number ofpixels used for the end result, which increases (e.g., maximizes) imagequality.

FIG. 5 illustrates an example implementation 500 of the sensor detectionarea shifted relative to the image circle for enhanced packagedetection. In the illustrated example, the image circle 134 is shownsubstantially as an ellipse (e.g., circle). However, the image circle134 may be any suitable shape, which is based on a shape and curvatureof the lens 130. The sensor detection area 132 captures a portion of theimage circle 134 projected onto the image sensor 128. In the illustratedexample, the image circle 134 includes an image of a person (e.g., theperson 110) standing in front of the electronic doorbell 122. A package(e.g., the package 114) is shown in the lower portion 408 of the imagecircle 134. The upper portion 406 of the image circle 134 captures aspace above the head and face 108 of the person 110 in the image anddoes not include any data associated with the person 110. After theimage sensor 128 captures a portion of the image that is bounded withinthe sensor detection area 132, distortion correction may be performed onthe captured image, an example of which is described with respect toFIG. 6 .

FIG. 6 illustrates an example implementation 600 of lens distortioncorrection in accordance with asymmetric camera sensor positioning in anelectronic doorbell. First, the image sensor 128 (from FIG. 1 andshifted relative to the lens 130) captures an image (e.g., image 602-1)including a portion of the image circle 134 and a portion (e.g., twolower corner areas 434) outside of the image circle 134 near the bottomof the sensor detection area 132. As illustrated, the captured image602-1 includes the lower portion 408 of the image circle 134 and blackcorner areas 604-1 corresponding to the two lower corner areas 434 ofthe sensor detection area 132. The captured image 602-1, however, doesnot include the upper portion 406 of the image circle 134 above the topedge 432 of the sensor detection area 132. The captured image 602-1 alsoincludes the package 114-1. In some aspects, the captured image 602-1may also include dark pixels (e.g., regions 606-1) or other artifactsnear the edge of the image circle 134, which is proximate to the blackcorner areas 604-1.

A keystone correction 608 is applied to the captured image 602-1 toprovide a keystoned image 602-2 for asymmetric correction. Because thecaptured image 602-1 includes vertical asymmetry due to the asymmetricpositioning of the image sensor 128 relative to the lens 130, thekeystone correction 608 is applied to remove some of the lensdistortion.

The keystoned image 602-2 is straightened to provide a straightenedimage 602-3, which corrects some of the lens distortion (e.g., verticalasymmetry) in the original captured image 602-1. Notice the person 110-2in the straightened image 602-3 is thinner (e.g., less distorted) thanthe person 110-1 in the original captured image 602-1. Additionally, theblack corner areas 604-1 have decreased in size to black corner areas604-2. The regions 606-1 have also decreased in size to regions 606-2.In some aspects, the package 114-1 may be altered due to thestraightening of the keystoned image 602-2. For example, the package114-2 in the straightened image 602-3 is smaller than the package 114-1in the original captured image 602-1. However, the package 114-2 issufficient in size to be identifiable as a package on the user'sdoorstep. In aspects, the keystone correction 608 includes applying atrapezoidal shape to the image, which is narrower at the top of theimage and wider at the bottom of the image. The keystone correctionreduces the risk of introducing artifacts towards the top of the imageduring subsequent dewarping of the image.

The edges of the straightened image 602-3 are dewarped to provide adewarped image 602-4. Dewarping the edges of the straightened image602-3 removes the black corner areas 604-2 from the bottom of the image.In aspects, the dewarping may cause some distortion to the package114-3, but the lens distortion of the person 110-3 is significantlyreduced. Further, some distortion may be acceptable because fullycorrecting the distortion may introduce artifacts into the image, whichdegrades the final image and diminishes the user experience. In anexample, the dewarped image 602-4 may include the regions 606-3 ofdarkened pixels that were proximate to the edge of the image circle 134.

The dewarped image 602-4 may be brightened to remove the darkened pixelsin the regions 606-3 and provide a final image 602-5. The final image602-5 includes a distortion-corrected image that includes the person110-3 (including the person's head and face) and the package 114-3 withno black or darkened pixels in the bottom corners (e.g., bottom cornerareas 610) of the final image 602-5. The final image 602-5 may then beprovided to the user's device (e.g., smartphone) for display.

Example Methods

FIG. 7 depicts an example method 700 for correcting lens distortioncaused by an electronic doorbell with asymmetric image sensorpositioning, in accordance with the techniques described herein. Inaspects, the method 700 may be performed by one or more processors ofthe electronic doorbell 122.

At 702, an image is captured or received having lens distortion at leastpartially based on an image sensor asymmetrically positioned relative toa lens. In aspects, the image sensor 128 is asymmetrically positionedrelative to the lens 130, resulting in a vertical offset of theimage-sensor optical axis 318 (e.g., optical center of the image sensor128) from the lens optical axis 316 (e.g., optical center of the lens130). Further, the image sensor 128 is oriented in a portraitorientation. As a result of the orientation and asymmetric positioningof the image sensor 128 relative to the lens 130, the sensor detectionarea 132 is vertically offset from a center of the image circle 134 toinclude a lower portion (e.g., the lower portion 408) of the imagecircle 134 and regions (e.g., the lower corner areas 434) beyond theboundary of image circle 134, which result in the captured image havingblack corner areas 604-1.

At 704, a keystone correction is applied to the captured image toprovide a keystoned image. For example, the keystone correction 608 maybe applied to the captured image 602-1 to provide the keystoned image602-2 usable to correct vertical asymmetry resulting from the asymmetricpositioning of the image sensor 128 relative to the lens 130.

At 706, the keystoned image is straightened to provide a straightenedimage. For example, the keystoned image 602-2 may be straightened tocorrect the vertical asymmetry and provide the straightened image 602-3.In an example, the straightening of the keystoned image 602-2essentially stretches the keystoned image 602-2 more at the top than atthe bottom of the image.

At 708, the straightened image is dewarped to provide a dewarped image.For example, the edges of the straightened image 602-3 are dewarped toprovide the dewarped image 602-4. The dewarping removes black areas(e.g., the black corner areas 604) corresponding to the two lower cornerareas 434 of the sensor detection area 132.

At 710, the dewarped image is brightened to provide a final image. Forexample, the dewarped image 602-4 is brightened to provide the finalimage 602-5, which has the lens distortion corrected and no blackcorners.

At 712, the final image is provided to computer-readable storage memory.For example, the final image 602-5 may be stored in local storage or inremote storage (e.g., online storage). Because the electronic doorbell122 is battery powered, the final image 602-5 may be wirelesslycommunicated to a remote data storage.

At 714, the final image is output to a user device (e.g., smartphone).In an example, the final image 602-5 may be provided to the user devicebased on a request for access to the local storage or the remotestorage. Optionally, the final image can be output to the user device at714 prior to, or simultaneously with, providing the final image tomemory storage at 712. Accordingly, the techniques described hereininclude keystone dewarping of a 3:4 portrait-configured image sensor toprovide a final image 602-5 having an enhanced vFOV with no blackcorners. These techniques enable enhanced package detection of a packageor other object located in the lower portion 408 of the image circle134, which may typically be outside of a conventional image sensor'sFOV. Accordingly, such a package may be detected based on the finalimage 602-5, and a notification can be provided to a user of anotherelectronic device associated with the electronic doorbell 122 to notifythe user that the package is present or, in some instances, that thepackage has been removed.

Example Computing System

FIG. 8 is a block diagram illustrating an example system 800 thatincludes an example device 802, which can be implemented as anyelectronic device (e.g., the electronic doorbell 122) that implementsaspects of asymmetric camera sensor positioning as described withreference to FIGS. 1-7 . The example device 802 may be any type ofcomputing device, client device, mobile phone, tablet, communication,entertainment, gaming, media playback, and/or other type of device.Further, the example device 802 may be implemented as any other type ofelectronic device that is configured for communication on a network,such as a thermostat, doorbell, hazard detector, camera, light unit,commissioning device, router, border router, joiner router, joiningdevice, end device, leader, access point, a hub, and/or other electronicdevices. The example device 802 can be integrated with electroniccircuitry, microprocessors, memory, input-output (I/O) logic control,communication interfaces and components, as well as other hardware,firmware, and/or software to communicate via the network. Further, thedevice 802 can be implemented with various components, such as with anynumber and combination of different components, as further describedbelow.

The device 802 includes communication devices 804 that enable wiredand/or wireless communication of device data 806, such as data that iscommunicated between the devices in a network, data that is beingreceived, data scheduled for broadcast, data packets of the data, datathat is synchronized between the devices, etc. The device data caninclude any type of communication data, as well as audio, video, and/orimage data that is generated by applications executing on the device.The communication devices 804 can also include transceivers for cellularphone communication and/or for network data communication. Thecommunication devices 804 can include wireless radio systems formultiple, different wireless communications systems. The wireless radiosystems may include Wi-Fi, Bluetooth™, Mobile Broadband, Bluetooth LowEnergy (BLE), and/or point-to-point IEEE 802.15.4. Each of the differentradio systems can include a radio device, antenna, and chipset that isimplemented for a particular wireless communications technology.

The device 802 also includes input/output (I/O) interfaces 808, such asdata network interfaces that provide connection and/or communicationlinks between the device, data networks (e.g., an internal network,external network, etc.), and other devices. The I/O interfaces can beused to couple the device to any type of components, peripherals, and/oraccessory devices. The I/O interfaces also include data input ports viawhich any type of data, media content, and/or inputs can be received,such as user inputs to the device, as well as any type of communicationdata, such as audio, video, and/or image data received from any contentand/or data source.

The device 802 includes a processing system 810 that may be implementedat least partially in hardware, such as with any type ofmicroprocessors, controllers, or the like that process executableinstructions. The processing system can include components of anintegrated circuit, a programmable logic device, a logic device formedusing one or more semiconductors, and other implementations in siliconand/or hardware, such as a processor and memory system implemented as asystem-on-chip (SoC). Alternatively or in addition, the device can beimplemented with any one or combination of software, hardware, firmware,or fixed logic circuitry that may be implemented with processing andcontrol circuits. The device 802 may further include any type of asystem bus or other data and command transfer system that couples thevarious components within the device. A system bus can include any oneor combination of different bus structures and architectures, as well ascontrol and data lines.

The device 802 also includes computer-readable storage memory 812, suchas data storage devices that can be accessed by a computing device, andthat provide persistent storage of data and executable instructions(e.g., software applications, modules, programs, functions, or thelike). The computer-readable storage memory described herein excludespropagating signals. Examples of computer-readable storage memoryinclude volatile memory and non-volatile memory, fixed and removablemedia devices, and any suitable memory device or electronic data storagethat maintains data for computing device access. The computer-readablestorage memory can include various implementations of random accessmemory (RAM), read-only memory (ROM), flash memory, and other types ofstorage memory in various memory device configurations.

The computer-readable storage memory 812 provides storage of the devicedata 806 and various device applications 814, such as an operatingsystem that is maintained as a software application with thecomputer-readable storage memory and executed by the processing system810. The device applications may also include a device manager, such asany form of a control application, software application, signalprocessing and control module, code that is native to a particulardevice, a hardware abstraction layer for a particular device, and so on.In this example, the device applications also include a smart-homeapplication 816 that implements aspects of the asymmetric camera sensorpositioning for enhanced package detection, such as when the exampledevice 802 is implemented as any of the electronic devices describedherein. The device 802 also includes a power source 818, such as thebattery 308. An alternating current (AC) power source may also be usedto charge the battery of the device.

In aspects, at least part of the techniques described for the electronicdoorbell 122 may be implemented in a distributed system, such as over a“cloud” 820 in a platform 822. The cloud 820 includes and/or isrepresentative of the platform 822 for services 824 and/or resources826.

The platform 822 abstracts underlying functionality of hardware, such asserver devices (e.g., included in the services 824) and/or softwareresources (e.g., included as the resources 826), and communicativelyconnects the example device 802 with other devices, servers, etc. Theresources 826 may also include applications and/or data that can beutilized while computer processing is executed on servers that areremote from the example device 802. Additionally, the services 824and/or the resources 826 may facilitate subscriber network services,such as over the Internet, a cellular network, or Wi-Fi network. Theplatform 822 may also serve to abstract and scale resources to service ademand for the resources 826 that are implemented via the platform, suchas in an interconnected device implementation with functionalitydistributed throughout the system 800. For example, the functionalitymay be implemented in part at the example device 802 as well as via theplatform 822 that abstracts the functionality of the cloud 820.

Some examples are described below:

An electronic doorbell comprising: a lens having a lens optical axis andproviding an image circle representing a scene captured by the lens, theimage circle having an upper portion and a lower portion separated by amiddle portion and arranged in a vertical stack, the upper portionlocated proximate to an upper edge of the image circle, the lowerportion located proximate to a lower edge of the image circle; and animage sensor having an sensor detection area, the image sensor being:positioned in a portrait orientation relative to the vertical stack ofportions of the image circle, the portrait orientation of the sensordetection area having a vertical FOV that is greater than a horizontalFOV; and vertically shifted relative to the lens toward the lowerportion of the image circle by an offset distance from the lens opticalaxis to enable the image sensor to capture an image of an object locatedin the lower portion of the image circle proximate to the lower edge ofthe image circle.

The image sensor may be configured in a 3:4 portrait orientation.

The sensor detection area may have a rectangular shape and the imagecircle may have an elliptical shape; and the sensor detection area mayinclude two upper corners located within the image circle and two lowercorners located outside of the image circle.

The image sensor may have an image sensor optical axis; and the imagesensor may be vertically shifted relative to the lens to have the imagesensor optical axis offset from the lens optical axis by the offsetdistance along a vertical axis of the lens.

The image sensor optical axis may be parallel to the lens optical axis.

The offset distance may be substantially within a range of 0.15millimeters to 0.35 millimeters.

The image circle represents the scene based on the lens having an angleof view of approximately 160°.

The portrait orientation and the vertical shift of the image sensorrelative to the lens may enable the image sensor to capture an area ofthe image circle corresponding to a vertical angle of view of the lensof approximately 130°.

The electronic doorbell may further comprise a processor configured toperform lens distortion correction on an image captured by the imagesensor to remove black areas in lower corner areas of the capturedimage.

The processor may be configured to perform lens distortion correctionby: applying a keystone correction to the captured image to provide akeystoned image usable to correct vertical asymmetry; and straighteningthe keystoned image to correct the vertical asymmetry and provide astraightened image.

The processor may be configured to perform the lens distortioncorrection by further dewarping the straightened image to remove theblack areas in the lower corner areas and provide a dewarped image.

The processor may be configured to perform the lens distortioncorrection by further brightening the dewarped image to provide a finalimage for display.

The image sensor may be positioned to have a midpoint of a bottom edgeof the sensor detection area located within the image circle.

A method for correcting lens distortion caused by an electronic doorbellwith asymmetric image sensor positioning, the method comprising:capturing an image using an image sensor of the electronic doorbellhaving: a sensor detection area oriented in a portrait orientation; andan image-sensor optical axis vertically shifted from a lens optical axisof a lens of the electronic doorbell, the sensor detection area havingupper corners located within an image circle projected onto the imagesensor by the lens and lower corners located outside of the imagecircle; applying a keystone correction to the captured image to providea keystoned image usable to correct vertical asymmetry resulting fromthe vertically shifted image-sensor optical axis; straightening thekeystoned image to correct the vertical asymmetry and provide astraightened image; dewarping the straightened image to remove blackareas corresponding to the lower corners of the sensor detection areaand provide a dewarped image; brightening the dewarped image to providea final image for display on an electronic device.

The method may further comprise: detecting a package located in a lowerportion of the image circle proximate to an edge of the image circlebased on the final image; and providing an indication to a user of theelectronic device that the package is present.

CONCLUSION

Although aspects of the asymmetric camera sensor positioning forenhanced package detection have been described in language specific tofeatures and/or methods, the subject of the appended claims is notnecessarily limited to the specific features or methods described.Rather, the specific features and methods are disclosed as exampleimplementations of the claimed asymmetric camera sensor positioning forenhanced package detection, and other equivalent features and methodsare intended to be within the scope of the appended claims. Further,various different aspects are described, and it is to be appreciatedthat each described aspect can be implemented independently or inconnection with one or more other described aspects.

What is claimed is:
 1. An electronic doorbell comprising: a lens havinga lens optical axis and providing an image circle representing a scenecaptured by the lens, the image circle having an upper portion and alower portion separated by a middle portion and arranged in a verticalstack, the upper portion located proximate to an upper edge of the imagecircle, the lower portion located proximate to a lower edge of the imagecircle; and an image sensor having a sensor detection area, the imagesensor being: positioned in a portrait orientation relative to thevertical stack of portions of the image circle, the portrait orientationof the sensor detection area having a vertical dimension that is greaterthan a horizontal dimension; and vertically shifted relative to the lenstoward the lower portion of the image circle by an offset distance fromthe lens optical axis to enable the image sensor to capture an image ofan object located in the lower portion of the image circle proximate tothe lower edge of the image circle.
 2. The electronic doorbell of claim1, wherein the image sensor is configured in a 3:4 portrait orientation.3. The electronic doorbell of claim 1, wherein: the sensor detectionarea has a rectangular shape and the image circle has an ellipticalshape; and the sensor detection area includes two upper corners locatedwithin the image circle and two lower corners located outside of theimage circle.
 4. The electronic doorbell of claim 1, wherein: the imagesensor has an image sensor optical axis; and the image sensor isvertically shifted relative to the lens to have the image sensor opticalaxis offset from the lens optical axis by the offset distance along avertical axis of the lens.
 5. The electronic doorbell of claim 4,wherein the image sensor optical axis is parallel to the lens opticalaxis.
 6. The electronic doorbell of claim 1, wherein the offset distanceis substantially within a range of 0.15 millimeters to 0.35 millimeters.7. The electronic doorbell of claim 1, wherein the image circlerepresents the scene based on the lens having an angle of view ofapproximately 160°.
 8. The electronic doorbell of claim 1, wherein theportrait orientation and the vertical shift of the image sensor relativeto the lens enables the image sensor to capture an area of the imagecircle corresponding to a vertical angle of view of the lens ofapproximately 130°.
 9. The electronic doorbell of claim 1, furthercomprising a processor configured to perform lens distortion correctionon an image captured by the image sensor to remove black areas in lowercorner areas of the captured image.
 10. The electronic doorbell of claim9, wherein the processor is configured to perform lens distortioncorrection by: applying a keystone correction to the captured image toprovide a keystoned image usable to correct vertical asymmetry; andstraightening the keystoned image to correct the vertical asymmetry andprovide a straightened image.
 11. The electronic doorbell of claim 10,wherein the processor is configured to perform the lens distortioncorrection by further dewarping the straightened image to remove theblack areas in the lower corner areas and provide a dewarped image. 12.The electronic doorbell of claim 11, wherein the processor is configuredto perform the lens distortion correction by further brightening thedewarped image to provide a final image for display.
 13. The electronicdoorbell of claim 1, wherein the image sensor is positioned to have amidpoint of a bottom edge of the sensor detection area located withinthe image circle.
 14. A method comprising: capturing an image using animage sensor of an electronic doorbell having: a sensor detection areaoriented in a portrait orientation; and an image-sensor optical axisvertically shifted from a lens optical axis of a lens of the electronicdoorbell, the sensor detection area having upper corners located withinan image circle projected onto the image sensor by the lens and lowercorners located outside of the image circle; applying a keystonecorrection to the captured image to provide a keystoned image usable tocorrect vertical asymmetry resulting from the vertically shiftedimage-sensor optical axis; straightening the keystoned image to correctthe vertical asymmetry and provide a straightened image; dewarping thestraightened image to remove black areas corresponding to the lowercorners of the sensor detection area and provide a dewarped image; andbrightening the dewarped image to provide a final image for display onan electronic device.
 15. The method of claim 14, further comprising:detecting a package located in a lower portion of the image circleproximate to an edge of the image circle based on the final image; andproviding an indication to a user of the electronic device that thepackage is present.
 16. The method of claim 15, further comprising:providing the final image to a storage memory; and outputting the finalimage to a user device based on a request for access to the storagememory.
 17. The method of claim 14, wherein the image sensor opticalaxis is parallel to the lens optical axis.
 18. The method of claim 14,wherein the portrait orientation and the vertical shift of the imagesensor relative to the lens enables the image sensor to capture an areaof the image circle corresponding to a vertical angle of view of thelens of approximately 130°.
 19. The method of claim 14, wherein theimage sensor is positioned to have a midpoint of a bottom edge of thesensor detection area located within the image circle.
 20. The method ofclaim 14, wherein the image sensor is configured in a 3:4 portraitorientation.